Structured Models for Audio Content Analysis Ph.D. Thesis Proposal

The ability to automatically analyze audio content is a key aspect of information retrieval systems that deal with multimodal files. The unprecedented growth of web-based user generated content-sharing platforms and their popularity has led to research efforts attempting to understand the content of such files. Typically, audio analysis research has focussed on some specific tasks – detection of specific types of sounds, classification of the content into categories , and summarizing the content of an audio file. These approaches involved working individually on small segments of audio using supervised methods to detect patterns of interest. The main hypothesis that drives this dissertation is that sound has its own language and structure and can be modeled using sequences of lower level units (which we refer to as acoustic unit descriptors). The lower level units may not carry semantic information individually, but the sequences or distribution of these units should capture semantic information. In this language for sounds, the lower level units alluded to would be analogous to the alphabet. Such a representation of sound using a discrete sequence lends itself naturally to a hierarchical structure, where sequences of these lower level units can be mapped to real events, that have clear semantic interpretations. Further, these event sequences themselves should carry information about the overall semantic content or class of the audio. Depending on the restrictions we enforce at various levels of this structure, we can use such structured models to classify or detect sound types, segment files as a sequence of semantically meaningful sound types, or predict associated sound classes. In this proposal, we first summarize our prior work that describes the process of learning of the lower level acoustic unit descriptors in an unsupervised manner from audio data. We demonstrate empirically that the learnt acoustic unit descriptors appear to capture semantic information, and that they can outperform other plausible semantically motivated schemes. We then discuss the proposed directions of research in this dissertation, including techniques that attempt to discover further structural relationships between sequences of these acoustic unit descriptors, or the event layer that lies above them. Our approach to discovering the hidden structure proposes to leverage the large amount of unlabeled and coarsely labeled data, using techniques inspired by semi-supervised and multi-instance learning approaches. The research pursued in this dissertation will demonstrate that hidden semantic structure can be automatically discovered from weakly-labeled audio data. The use of such semantically informed features would enable audio analysis to improve significantly over the state-of-the-art.